R for Data Science - Data Science

What is data science? A buzzword with a blurry definition…

• Anything you do with (Big) data to extract some information (about a specific field)?
• A typical data science project workflow:

R for Data Science - Big Data

What is big data?
Another buzzword with a blurry definition?

• Or with Dan Ariely’s often cited words:
  Big data is like teenage sex;
  - everyone talks about it
  - nobody really knows how to do it,
  - everyone thinks everyone else is doing it,
  - so everyone claims they are doing it.

R for Data Science - Big Data

• “Big data typically refers to data on the scale of terabytes (10 to the 12th power) and petabytes (10 to the 15th power).” - datascience@berkeley

• A petabyte is a million gigabytes (GB).

• A better description:
  “In traditional analysis, the development of a statistical model takes more time than the calculation by the computer. When it comes to Big Data this proportion is turned upside down. Big Data comes into play when the CPU time for the calculation takes longer than the cognitive process of designing a model.” - Hadley Wickham

• R can easily handle data sizes up to 100s of MBs. And additional R packages can help with sizes up to 10s of GBs, or with even larger data sizes depending on the “big data problem”.

R for Data Science - Big Data

Is your “big data” actually big data?

• Small data in disguise:
  Although the complete data might be big, the data needed to answer the question of interest can be small that fits in memory.

• A large number of small data :
  Although the data might be big, it might comprise of many small data problems. Each individual problem might fit in memory, but you have millions of them. In that case systems like Hadoop or Spark can help and R packages like sparklyr, rhipe, and ddr make this process easy.

R for Data Science - Why R is so awesome

R offers great tools for the entire Data Science process

• It’s all open source!!

R for Data Science - Why R is so awesome

R for Data Science - Why R is so awesome

Or interactive plots:

library(plotly)
p <- iris %>% 
    ggplot(aes(x=Sepal.Length, y=Sepal.Width, color=Species)) +
    geom_point() +
    geom_smooth()

ggplotly(p)    

R for Data Science - Why R is so awesome

R community #rstats on Twitter

Column headers are values, not variable names - pew example

• Example dataset: pew.csv
  
• It comes from a report produced by the Pew Research Center (which produces many reports that contain datasets in this format).
  
• Explores the relationship between income and religion in the US.

library(tidyverse)
pew <- read_csv("data/pew.csv")
head(pew)

• This dataset has three variables. What are they?
• religion, income and frequency.
  
• To tidy it, we need to gather the non-variable columns into a two-column key-value pair.

• This action is often described as making a wide dataset long (or tall) or called “melting”.

• We can easily gather the columns with the gather() function.
  
• gather has 4 main arguments:
  • data: The data frame to gather
  • key: Name of the key column, which will contain the collapsed column names, In this case the column name for this key column would be income
  • value: Name of the value column, which will populate the gathered values. In this case, it’s n
  • specify the columns to gathered: bare column names of the columns that will be collapsed. Here, it’s every column except religion.

gather(data=pew, key=income, value=n, -religion)

If we would use the pipe %>% same code would look like this:

pew %>%
    gather(key=income, value=n, -religion) %>%
    head

This form is tidy because each column represents a variable and each row represents an observation, which is in this case a demographic unit corresponding to a combination of religion and income.

Column headers are values, not variable names - billboard

This format is also used to record regularly spaced observations over time.

• Example dataset: billboard.csv
  
• Billboard dataset shown below records the date a song first entered the billboard top 100 and the rank of the song in each week after that up to 75 weeks.
  
• It has variables for artist, track, date.entered, rank and week, plus the columns from wk1 to wk75.
  This form of storage is useful for data entry, but not useful for plotting or modelling!

billboard <- read_csv("data/billboard.csv")
billboard[1:3, 1:10]

Again, to tidy this dataset, we first gather together all the wk columns. The column names give the week and the values are the ranks:

billboard2 <- billboard %>% 
  gather(week, rank, wk1:wk76, na.rm = TRUE)
billboard2 %>% head

Here we use na.rm to drop any missing values from the gather columns. In this data, missing values represent weeks that the song wasn’t in the charts, so can be safely dropped.

In this case it’s also nice to do a little cleaning, converting the week variable to a number, and figuring out the date corresponding to each week on the charts:

billboard3 <- billboard2 %>%
  mutate(
    week = parse_number(week),
    date = as.Date(date.entered) + 7 * (week - 1)) %>%
  select(-date.entered)
billboard3 %>% head

Finally, it’s always a good idea to sort the data. We could do it by artist, track and week:

billboard3 %>% arrange(artist, track, week)

Or by date and rank:

billboard3 %>% arrange(date, rank) %>% head

Multiple variables stored in one column

After gathering columns, the key column is sometimes a combination of multiple underlying variable names.
>- Example dataset: tb (tuberculosis) >- From the World Health Organisation, and records the counts of confirmed tuberculosis cases by country, year, and demographic group.
>- The demographic groups are broken down by sex (m, f) and age (0-14, 15-25, 25-34, 35-44, 45-54, 55-64, unknown).

tb <- read_csv("data/tb.csv")
tb[1:3, 1:10]

First we gather up the non-variable columns:

tb2 <- tb %>% 
  gather(key = demo, value = n, -iso2, -year, na.rm = TRUE)
tb2 %>% head

Column headers in this format are often separated by a non-alphanumeric character (e.g. ., -, _, :), or have a fixed width format, like in this dataset. separate() makes it easy to split a compound variables into individual variables. You can either pass it a regular expression to split on (the default is to split on non-alphanumeric columns), or a vector of character positions. In this case we want to split after the first character:

tb3 <- tb2 %>% 
  separate(demo, c("sex", "age"), 1)
tb3 %>% head

Storing the values in this form resolves a problem in the original data. We want to compare rates, not counts, which means we need to know the population.
In the original format, there is no easy way to add a population variable.
It has to be stored in a separate table, which makes it hard to correctly match populations to counts.
In tidy form, adding variables for population and rate is easy because they’re just additional columns.

Variables are stored in both rows and columns

The most complicated form of messy data occurs when variables are stored in both rows and columns.

• Example dataset: weather
  
• Daily weather data from the Global Historical Climatology Network for one weather station (MX17004) in Mexico for five months in 2010.

weather <- read_csv("data/weather.csv")
weather[1:3, 1:15]

• It has variables in individual columns (id, year, month), spread across columns (day, d1-d31) and across rows (tmin, tmax) (minimum and maximum temperature).
  
• Months with fewer than 31 days have structural missing values for the last day(s) of the month.

To tidy this dataset we first gather the day columns:

weather2 <- weather %>%
  gather(key = day, value = value, d1:d31, na.rm = TRUE)
weather2 %>% head

At this point I am removing the missing values with na.rm.
This is ok because we know how many days are in each month and can easily reconstruct the explicit missing values.

We’ll also do a little cleaning:

weather3 <- weather2 %>% 
  mutate(day = parse_number(day)) %>%
  select(id, year, month, day, element, value) %>%
  arrange(id, year, month, day)
weather3 %>% head

• This dataset is mostly tidy, but the element column is not a variable; it stores the names of variables.
  
• Fixing this requires the spread operation. This performs the inverse of gathering by spreading the element and value columns back out into the columns:

weather4 <- weather3 %>% 
    spread(key = element, value = value)
weather4 %>% head

This form is tidy: there’s one variable in each column, and each row represents one day.

Multiple types in one table - billboard

• Datasets often involve values collected at multiple levels, on different types of observational units.
  
• During tidying, each type of observational unit should be stored in its own table. This is closely related to the idea of database normalization, where each fact is expressed in only one place.
  
• It’s important because otherwise inconsistencies can arise.

The billboard dataset actually contains observations on two types of observational units: the song and its rank in each week. This manifests itself through the duplication of facts about the song: artist, year and time are repeated many times.

billboard3 %>%
    head

This dataset needs to be broken down into two pieces:
- a song dataset which stores artist, song name and time
- and a ranking dataset which gives the rank of the song in each week.

We first extract a song dataset:

song <- billboard3 %>% 
  select(artist, track, year, time) %>%
  unique() %>%
  mutate(song_id = row_number())
song %>% head

Then use that to make a rank dataset by replacing repeated song facts with a pointer to song details (a unique song id):

rank <- billboard3 %>%
  left_join(song, c("artist", "track", "year", "time")) %>%
  select(song_id, date, week, rank) %>%
  arrange(song_id, date)
rank %>% head

• You could also imagine a week dataset which would record background information about the week, maybe the total number of songs sold or similar “demographic” information.

• Although this is useful for tidying and eliminating inconsistencies, most analytic tools usually also requires merging the datasets back into one table.

One type in multiple tables

• It’s also common to find data values about a single type of observational unit spread out over multiple tables or files.
  
• These tables and files are often split up by another variable, so that each represents a single year, person, or location. As long as the format for individual records is consistent, this is an easy problem to fix:

1. Read the files into a list of tables.

2. For each table, add a new column that records the original file name (the file name is often the value of an important variable).

3. Combine all tables into a single table.

• map() function from purr makes this straightforward in R.
  
• The following code generates a vector of file names in a directory (data/) which match a regular expression (iris* - any file starts with “iris”).
  
• Next we read in each file and row bind them into a data frame with map_df()

(paths <- dir("data", pattern = "iris*", full.names = TRUE))
#> [1] "data/iris1.txt" "data/iris2.txt" "data/iris3.txt"
map_df(paths, read_tsv) %>% head

• If you would like to keep the file name associated with data one way would be to write a short function and again apply that to each element of paths.
  
• Let’s call our function read_files(), it will read the data in and then add a new column file with the name of the file.
• Also I use basename function to get the bare name of the file without the full path.

basename(paths)
#> [1] "iris1.txt" "iris2.txt" "iris3.txt"
read_files <- function(x) {
    read_tsv(x) %>%
        mutate(file=basename(x))
}
map_df(paths, read_files) %>% head